Systems and Methods for Reading Comprehension for a Question Answering Task

ABSTRACT

Embodiments described herein provide a pipelined natural language question answering system that improves a BERT-based system. Specifically, the natural language question answering system uses a pipeline of neural networks each trained to perform a particular task. The context selection network identifies premium context from context for the question. The question type network identifies the natural language question as a yes, no, or span question and a yes or no answer to the natural language question when the question is a yes or no question. The span extraction model determines an answer span to the natural language question when the question is a span question.

PRIORITY APPLICATION DATA

This application claims priority to U.S. Provisional Application No.62/851,048 filed on May 21, 2019 and entitled “Systems and Methods forReading Comprehension for a Question Answering Task,” which isincorporated by reference in its entirety.

TECHNICAL FIELD

The disclosure relates generally to training and use of machine learningsystems and more specifically to a reading comprehension model for aquestion answering task.

BACKGROUND

Pretrained bidirectional language models (e.g., Bidirectional EncoderRepresentations from Transformers or BERT models) have been used in manynatural language processing tasks, such as question answering,classification, and/or the like. However, for some more challengingquestions, the performance of such models may be unsatisfactory comparedto human-level accuracy and techniques for improving BERT models aredesired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a simplified diagram of a computing device implementing aquestion answering system, according to some embodiments.

FIGS. 2A-2C are graphs that illustrate sums of attention weights ontokens at different layers of a BERT model, according to someembodiments.

FIG. 3 is a graph that illustrates question-paragraphs pairs in aHotpotQA dataset, according to some embodiments.

FIGS. 4A-4B are graphs that illustrate the scores for attended tokens ineach layer of a BERT model, according to some embodiments.

FIG. 5 is a block diagram of a pipeline question answering module,according to some embodiments.

FIG. 6 is a simplified logic flow diagram illustrating a method fordetermining an answer to a natural language question, according to someembodiments.

In the figures, elements having the same designations have the same orsimilar functions.

DETAILED DESCRIPTION

In view of the need to improve accuracy in question answering models,the embodiments describe a pipeline question answering model. Thepipeline question answering model includes a context selection modulethat selects premium context from a context, such as a text or adocument, using context and natural language question as input. Thepipeline question answering model also includes a yes-no span modulethat identifies a type of a natural language question, such as a yes,no, or span question using the premium context and natural languagequestion as input. Additionally, the pipeline question answering modelincludes a span extraction module that identifies an answer span fromthe premium context for the span question. In some embodiments, thecontext selection module, the yes-no span module, and the spanextraction module may be implemented sequentially or in parallel as BERTmodels, each trained to perform a particular task.

As used herein, the term “network” may comprise any hardware orsoftware-based framework that includes any artificial intelligencenetwork or system, neural network or system and/or any training orlearning models implemented thereon or therewith.

As used herein, the term “module” may comprise hardware orsoftware-based framework that performs one or more functions. In someembodiments, the module may be implemented on one or more neuralnetworks.

FIG. 1 is a simplified diagram of a computing device 100 forimplementing a question answering system according to some embodiments.As shown in FIG. 1, computing device 100 includes a processor 110coupled to a memory 120. Operation of computing device 100 is controlledby processor 110. And although computing device 100 is shown with onlyone processor 110, it is understood that processor 110 may berepresentative of one or more central processing units, multi-coreprocessors, microprocessors, microcontrollers, digital signalprocessors, field programmable gate arrays (FPGAs), application specificintegrated circuits (ASICs), graphics processing units (GPUs) and/or thelike in computing device 100. Computing device 100 may be implemented asa stand-alone subsystem, as a board added to a computing device, and/oras a virtual machine.

Memory 120 may be used to store software executed by computing device100 and/or one or more data structures used during operation ofcomputing device 100. Memory 120 may include one or more types ofmachine readable media. Some common forms of machine readable media mayinclude floppy disk, flexible disk, hard disk, magnetic tape, any othermagnetic medium, CD-ROM, any other optical medium, punch cards, papertape, any other physical medium with patterns of holes, RAM, PROM,EPROM, FLASH-EPROM, any other memory chip or cartridge, and/or any othermedium from which a processor or computer is adapted to read.

Processor 110 and/or memory 120 may be arranged in any suitable physicalarrangement. In some embodiments, processor 110 and/or memory 120 may beimplemented on a same board, in a same package (e.g.,system-in-package), on a same chip (e.g., system-on-chip), and/or thelike. In some embodiments, processor 110 and/or memory 120 may includedistributed, virtualized, and/or containerized computing resources.Consistent with such embodiments, processor 110 and/or memory 120 may belocated in one or more data centers and/or cloud computing facilities.

In some examples, memory 120 may include non-transitory, tangible,machine readable media that includes executable code that when run byone or more processors (e.g., processor 110) may cause the one or moreprocessors to perform the methods described in further detail herein.For example, as shown, memory 120 includes instructions for QA module130 that may be used to implement and/or emulate the systems and models,and/or to implement any of the methods described further herein. In someexamples, QA module 130 may be used to receive and handle the input of anatural language question 140. In some embodiments, QA module 130 maygenerate answer 150 for natural language question 140 using context 160.Context 160 may be a dataset, document, text, one or more paragraphs,etc. In some embodiments, QA module 130 may also handle the iterativetraining and/or evaluation of a system or model used for questionanswering tasks.

In some embodiments, context 160 may be a document, text or paragraph(s)included in a SQuAD or HotpotQA dataset. The SQuAD (Stanford QuestionAnswering Dataset) may be a reading comprehension dataset that containshuman-annotated answer spans for pairs that include a question and asingle paragraph in each pair. An example pair in a SQuAD that includesa question Q and paragraph P is shown in Table 1, replicated below. Ananswer A (“France”) to a question Q (“In what county is Normandylocated?”) in the question-paragraph pair is also shown in Table 1.

TABLE 1 HotpotQA SQuAD Bridge Comparison Q In what country is Normandy QIn what year was Kasper Schmeichel's Which band was formed firstlocated? father born? Killing Joke or Acceptance? P The Normans (Norman:Nour- P1 Kasper Peter Schmeichel is a Danish Killing Joke are an Englishrock mands; French: Normands; professional footballer. He is the son ofband formed in October 1978 in Latin: Normanni) were the peo- PeterSchmeichel. Notting Hill, London, England. ple who in the 10th and 11thcenturies gave their name io Normandy, a region in France. P2 PeterBolesaw Schmeichel (born 18 Acceptance is an American rock Nov. 1963) isa professional foot- band from Seattle, Washington, baller who played asa goalkeeper. formed in 1998. A France A 1963 Killing Joke

In some embodiments, the HotpotQA dataset may be a reading comprehensiondataset that may provide an answer to a question using pairs thatinclude a question and multiple paragraphs in each pair. The number ofparagraphs in each pair may be ten but may also be configured to includea different number of paragraphs. In some embodiments, a question in theHotpotQA dataset may be characterized into a bridge question or acomparison question. A bridge question may have one or more hops betweenthe paragraphs in the question-paragraphs pair to determine an answerthe question. A comparison question may compare entities in the one ormore paragraphs in the question-paragraphs pair. Example bridge questionin Table 1 is “In what year was Kasper Schmeichel's father born?” andexample comparison question is “Which band was formed first Killing Jokeor Acceptance?”

In some embodiments, one or more paragraphs in the question-paragraphspair in the HotpotQA dataset may be annotated to include supportingfacts. These paragraphs may be referred to as premium context or “gold”paragraphs. The “gold” paragraphs may be top-2 paragraphs based on theprobability values that may include an answer to a question orparagraphs that have the probability of including an answer to aquestion above a configurable threshold. Table 1, above, illustrates anexample question-paragraphs pair in a HotpotQA dataset, that includes aquestion Q and paragraphs P1 and P2. In a non-limiting embodiment,paragraphs P1 and P2 are “gold” paragraphs that determine an answer A tothe question Q.

In some embodiments, QA module 130 may be a BERT (Bidirectional EncoderRepresentations from Transformers) network. The BERT network may be aneural network that has multiple layers, where each layer has one ormore bi-directional transformer encoders. Some BERT networks may have 12or 24 layers. Additionally, the transformer encoders may have vector(s)of weights that may manipulate input and generate output. With anexception of the first and last layers, the encoders at each layer maybe connected to one or more encoders of the preceding layer and generatean output that is an input to the one or more encoders at the subsequentlayer. The first layer in the network may receiving input, e.g., naturallanguage question 140 and context 160, while the last layer may outputan answer, e.g., answer 150 to natural language question 140. At eachlayer, the encoders may generate an output by applying a vector ofweights to the input. Typically, the weights may be determined bytraining the QA module 130.

As discussed above, QA module 130 may receive natural language question140 and context 160 as input. Natural language question 140 and context160 may be broken up into tokens, e.g., each word in natural languagequestion 140 may be a token. Once QA module 130 receives tokens fornatural language question 140 and context 160, QA module 130 maydetermine answer 150 to natural language question 140 by passing thetokens from natural language question 140 and context 160 through thelayers in QA module 130 where the weights of the transformer encodersact on the tokens. At a final layer of QA module 130, the encoders maygenerate answer 150 to natural language question 140.

In some embodiments, QA module 130 may be trained to determine answer150 for natural language question 140 using context 160. Duringtraining, QA model 130 may receive natural language question 140 andcontext 160 and train the weights in the vectors to store values thatcause QA module 130 to output a known answer 150. Once trained, QAmodule 130 may receive natural language question 140 and context 160 andgenerate answer 150.

In some embodiments, QA module 130 may include one or more attentionheads in each layer. In one example, a 12-layer model may include 12attention heads, one attention head per layer. In another example, a24-layer model may include 16 attention heads, where some layers mayinclude one or more attention heads. Attention heads, like encoders, mayreceive tokens included in natural language question 140 or context 160and apply the matrices or vectors that may generate attention weightsfor each token. Attention weights may indicate importance of each tokenin the sequence of tokens in relationship with other tokens in thesequence. For example, with respect to question “In what country isNormandy located?” the attention heads in QA module 130 may generate anattention weight for tokens “in,” “what,” “country,” “is,” “Normandy,”and “located,” and use the weights to determine an importance of thesetokens to each other.

In some embodiments, memory 120 may include a layer analyzer 170. Layeranalyzer 170 may be a software or hardware component that analyzes QAmodule 130 and determines the portions of context 160 that QA module 130may read when QA module 130 determines answer 150 for natural languagequestion 140. To determine the portions of context 160 that QA module130 may read, layer analyzer 170 may analyze attention heads at eachlayer. For example, for each layer, layer analyzer 170 may determine asum of the weights of the attention heads for each token. Layer analyzer170 may then map the summed weights to the tokens in natural languagequestion 140 and context 160. The higher the attention weights for atoken, the more emphasis QA module 130 may place on the token ascompared to other tokens. In some embodiments, when context 160 is fromHotpotQA dataset, layer analyzer 170 may analyze tokens from the “gold”paragraphs in context 160, and not other paragraphs. As discussed above,this is because the “gold” paragraphs likely include answer 150 tonatural language question 140.

In some embodiments, layer analyzer 170 may analyze the summed attentionweights for the tokens at each layer. Layer analyzer 170 may use thesums to determine the portions of natural language question 140 andcontext 160 that QA module 130 reads at each layer.

FIGS. 2A-2C are graphs that illustrate the mapping of the summedattention weights for tokens at layer two, layer nine, and layer twelveof a 12-layer QA model. In FIGS. 2A-2C the sum of attention weights fortokens for context 160 is shown on the “x” axis and for tokens fromnatural language question 140 is shown on the “y” axis. FIGS. 2A-2C alsoinclude a legend 210 that maps the summed attention weight to a color,where the higher the sum of the attention weights, the lighter is thecorresponding color and the lower the sum of the attention weights, thedarker is the corresponding color.

In some embodiments, layer analyzer 170 may determine that at the lowerlayers QA module 130 may focus on lexical or semantic matching betweenthe tokens from natural language question 140 and tokens from context160. FIG. 2A illustrates that at layer 2, QA module 130 focuses on thetokens “Britain” from natural language question 140 and context 160because these tokens have the lightest color and hence the highest scorein FIG. 2A. Also, QA module 130 focuses on token “Britain” from naturallanguage question 140 and token “British” from context 160, and tokens“common” from natural language question 140 and context 160 becausethese tokens also have high scores as shown in the mapping in legend210.

In some embodiments, layer analyzer 170 may determine that at higherlayers QA module 130 may focus important terms, such as answer phrasesor key words that may later be used to determine answer 150. FIG. 2Billustrates that at layer 9 of the 12-layer QA module 130, the mappingillustrates that QA module 130 focuses on the tokens “Tony” and “Ben”from context 160 shown in the “x” axis since those tokens have highscores as compared to other tokens. The tokens “Tony” and “Ben” aremapped to multiple tokens in the “y” axis that shows tokens from naturallanguage question 140. This indicates that tokens “Tony” and “Ben” maybe important terms in context 160 that may be associated with answer 150to natural language question 140.

As illustrated in FIG. 2C, at a final layer of the 12-layer QA module130, layer analyzer 170 may determine that QA module 130 focuses onpossible answer candidates for answer 150. As illustrated in FIG. 2C,token “47” has the highest weighted attention score and answer 150 tonatural language question 140.

Accordingly, based on FIGS. 2A-2C, layer analyzer 170 may determine thatQA module 130 may initially place importance on answer types andsemantic relationships at lower layers before finalizing answercandidates for answer 150 at the higher layers.

In some embodiments, layer analyzer 170 may also analyzer the top Kattended words (represented as tokens) in the layers of QA module 130. Kmay be a positive integer. To analyze the top K attended words, QAmodule 130 may receive context 160 where key entities are annotated formulti-sentence reasoning and comprehension. FIG. 3 is a chart thatillustrates three question-paragraphs pairs in a HotpotQA dataset. Asdiscussed above, the paragraphs may be “gold” paragraphs that arepremium context in context 160 for natural language question 140. Theparagraphs are annotated with annotations 310, 320, and 330. Annotations310 may indicate key entity and key phrases in the paragraphs,annotations 320 may indicate answers, and annotations 330 may indicatewords that overlap in the paragraphs.

In some embodiments, layer analyzer 170 may evaluate a score at eachlayer for the K attended words in natural language question 140, context160, and answer 150. Layer analyzer 170 may determine score as follows:

$\begin{matrix}{S_{d}^{q,q,e} = {\frac{1}{N}{\sum_{i = 1}^{N}\frac{c\left( x_{i} \right)}{\min \left( {{l\left( x_{i} \right)},K} \right)}}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

where x_(i) is either the i-th {natural language question 140, answer150, or key phrases}, l(x_(i)) is the length for x_(i), and c(x_(i))counts how many words in x_(i) are covered by the top-K list. FIGS. 4Aand 4B are diagrams that illustrate the scores for the 12 layers (layers0 through 11) of a 12-layer QA module 130 for the words K in naturallanguage question 150 (Q) and answer 140 (A) for HotpotQA and SQuADdatasets respectively. FIG. 4A also illustrates annotated key entity (E)that is used to bridge multi-sentence or paragraph reasoning. FIGS. 4Aand 4B also include a legend 410 that maps the scores to colors, withhigher scores having lighter colors, according to some embodiments.

In some embodiments, FIG. 4A illustrates that QA module 130 receivescontext 160 from HotpotQA dataset and first captures the interactionswithin natural language question 140 as illustrated by the higher scoresfor K words in question Q in layers 1-5. Next, QA module 130 graduallycaptures the key entities E as illustrated by the scores for K words inlayers 5-9. Next, QA module 130 finalizes answer 140 as illustrated bythe higher scores for K words in layers 9-11 for answer A.

In some embodiments, FIG. 4B illustrates that QA module 130 receivescontext 160 from SQuAD dataset and first captures the interactionswithin natural language question 140 as illustrated in layers 1-5 thathave relatively high scores for K words in question Q. Then, QA module130 gradually finalizes answer 150 as illustrated by the relatively highscores for K words in layers 9-11 for answer A.

In some embodiments, memory 120 also includes instructions for apipeline QA module 180 that may be used to implement and/or emulate thesystems and models, and/or to implement any of the methods describedfurther herein Like QA module 130, pipeline QA module 180 may receivenatural language question 140 and context 160 as input and determinesanswer 150. In some embodiments, pipeline QA module 180 may includemultiple QA modules that may be the same or different from QA module130, but that are further trained or “fine-tuned” to execute aparticular task. To fine-tune QA module 130, an extra one or more layersmay be added to QA module 130 and the transformer encoders in the addedlayer(s) may be trained to perform a particular task. In someembodiments, pipeline QA module 180 may include a context selectionmodule 510, a yes-no span module 520, and a span extraction module 530,which may be serially connected or connected in other manners, and whichare may be variants of QA module 130 discussed above.

In some embodiments, context selection module 510 may select premiumcontext (e.g. “gold” paragraphs) from context 160. As discussed above,premium context may include answer 150 to natural language question 140.For example, context selection module 510 may receive tokens for naturallanguage question 140 and context 160 that includes, e.g. ten paragraphsand may select premium context such as “gold” paragraphs from context160. To select the premium context, context selection module 510 may usea BERT model with a classifier. The classifier may be included in thelast or extra classification layer that classifies, for example, eachparagraph in context 160 as premium context or not as premium context.The classification may be a binary classification where “0” or “1” mayindicate the premium context. In some embodiments, context selectionmodule 510 may classify the premium context as the top two paragraphsthat have the highest probability that they are premium paragraphs. Inother embodiments, context selection module 510 may classify premiumcontext as paragraphs that have a probability that the paragraphs arepremium above a configurable or predefined probability threshold.

In some embodiments, to classify context 160 into premium andnon-premium context, context selection module 510 may be trained orpre-trained and fine-tuned. For example, context selection module 510may be a pre-trained BERT model where the classification layer isfine-tuned using context 160 that includes known premium context andnatural language question 140 until context selection module 510identifies the premium context as premium.

In some embodiments, yes-no span module 520 may be a question typemodule that determines whether natural language question 140 is a “yes,”“no,” or “span” question. The “yes” or “no” question may have an answerthat is a “yes” or a “no.” The “span” question is a question that may beanswered using a sentence or an answer span. The “span” question mayalso be a question that is not a “yes” or “no” question. In someembodiments, yes-no span module 520 may use a BERT three-classclassification model. The BERT three-class classification model may be aBERT model with an extra layer that uses three-way classification toclassify natural language question 140 as a “yes,” “no,” or “span”question. The input to yes-no span module 520 may be natural languagequestion 140 and the premium context which, in some embodiments, may bethe output of context selection module 510. The output of yes-no spanmodule 520 may be a classifier that identifies natural language question140 as a “yes,” “no,” or “span” question. To classify natural languagequestion 140 as a “yes,” “no,” or “span” question, context selectionmodule 510 may be trained or pre-trained and fine-tuned. For example,yes-no span module 520 may be a pre-trained BERT model where thethree-way classification layer is fine-tuned using context 160 and known“yes,” “no,” and “span” natural language questions until yes-no spanmodule 520 correctly classifies the natural language questions as “yes,”“no,” or “span” questions.

In the embodiments where yes-no span module 520 classifies naturallanguage question 140 as a “yes” or “no” question, pipeline QA module180 determines that answer 150 to natural language question 140 is “yes”or “no.” In this case, pipeline QA module 180 may adopt the classifierfrom yes-no span module 520 as answer 150. On the other hand, in theembodiments where yes-no span module 520 classifies natural languagequestion 140 as a “span” question, pipeline QA module 180 may use a spanextraction module 530 to determine answer 150.

In some embodiments, span extraction module 530 may determine an answerspan for natural language question 140 that yes-no span module 520identified as a “span” question. Span extraction module 530 may beconfigured to implement a BERT classification model to extract an answerspan that is answer 150. In some embodiments, the input to spanextraction module 530 may be natural language question 140 and thepremium context which may be the output of context selection module 510.In some embodiments, span extraction module 530 may include a BERT modelthat may be trained using a SQuAD dataset.

In some embodiments, QA module 130, layer analyzer 170, pipeline QAmodule 180, context selection module 510, yes-no span module 520, andspan extraction module 530, may be implemented using hardware, software,and/or a combination of hardware and software.

FIG. 6 is a simplified logic flow diagram illustrating a method 600 forgenerating an answer to a natural language question using a pipeline QAmodule, according to some embodiments. One or more of the processes610-650 of method 600 may be implemented, at least in part, in the formof executable code stored on non-transitory, tangible, machine-readablemedia that when run by one or more processors may cause the one or moreprocessors to perform one or more of the processes 610-650. In someembodiments, method 600 may correspond to the method used by thepipeline QA module 180 to perform different question answering tasks.

At a process 610, a context and a natural language question arereceived. For example, pipeline QA module 180 may receive context 160and natural language question 140.

At a process 620, premium context is determined from the context andnatural language question 140. For example, context selection module 510may include a BERT classification model that classifies portions (e.g.paragraphs) of context 160 as premium or non-premium context.

At process 630, the type of the natural language question is determined.For example, yes-no span module 520 may determine whether naturallanguage question 140 is a “yes,” “no,” or “span” question using thecontext paragraphs determined in process 620 and natural languagequestion 140. As discussed above, yes-no span module 520 may be a BERTthree-class classification model. When yes-no span module 520 determinesthat natural language question 140 is a “yes” or “no” question, method600 proceeds to process 640. When yes-no span module 520 determines thatnatural language question 140 is a “span” question, method 600 proceedsto process 650.

At process 640, an answer to a natural language question is outputted.Answer 150 may be a classification of yes-no span module 520 that is a“yes” or “no” answer to natural language question 140.

At a process 650, an answer is determined. For example, span extractionmodule 530 may determine an answer span for natural language question140 from context paragraphs determined in process 620. The answer spanis answer 150.

As illustrated in Table 2, below, pipeline QA module 180 may outperforma conventional BERT model, a BERT model with a sliding window (thesliding window receives a maximum number of paragraphs from the contextat a time and then slides to receive another maximum number ofparagraphs from the context), and a dynamic feature generation network(DFGN) that combines a BERT model with convolutional neural networks fordetermining an answer as compared to human accuracy when answering aquestion.

TABLE 2 HotpotQA P Method F1 EM EM Human 91.40 83.60 — BERT (base,sliding window) 60.49 47.25 — P-BERT (base, threshold) 69.90 56.39 71.24P-BERT (base) 70.16 57.80 89.39 P-BERT (large) 74.60 60.58 89.39BaseTine 58.28 44.44 — DFGN 69.23 55.42 —Table 2 also illustrates the results from three different pipeline QAmodules 180 illustrated in Table 2. The three pipeline QA modules 180are P-BERT module (base, threshold), P-BERT module (base) and P-BERTmodule (large). The P-BERT module (base, threshold) has contextselection module 510 that selects premium paragraphs from context 160that have a probability threshold of 0.1. P-BERT module (base) includesa 12-layer BERT model(s), and P-BERT (large) includes a 24-layer BERTmodel(s). Further, Table 2 illustrates results for context 160 that ispart of a HotpotQA dataset that includes multiple paragraphs and resultsfor context 160 that is part of a single paragraph dataset, such asSQuAD (shown as P in Table 2).

As Table 2 illustrates, BERT module (base, threshold), P-BERT module(base) and P-BERT module (large) show improvements in determining answer150 from conventional BERT and DFGN models. Table 2 also illustratesthat the P-BERT models may or may not use a probability threshold whenselecting premium context. This is because the accuracy of answer 150determined using BERT module (base, threshold) and P-BERT module (base)is approximately the same. Table 2 model further illustrates that a BERTmodel that uses a sliding window has results that are below those of theP-BERT modules which illustrates that the sliding window may or may notselect premium context or “gold” paragraphs when determining an answer.

Some examples of computing devices, such as computing device 100 mayinclude non-transitory, tangible, machine readable media that includeexecutable code that when run by one or more processors (e.g., processor110) may cause the one or more processors to perform the processes ofmethod 600. Some common forms of machine readable media that may includethe processes of method 600 are, for example, floppy disk, flexibledisk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, anyother optical medium, punch cards, paper tape, any other physical mediumwith patterns of holes, RAM, PROM, EPROM, FLASH-EPROM, any other memorychip or cartridge, and/or any other medium from which a processor orcomputer is adapted to read.

This description and the accompanying drawings that illustrate inventiveaspects, embodiments, implementations, or applications should not betaken as limiting. Various mechanical, compositional, structural,electrical, and operational changes may be made without departing fromthe spirit and scope of this description and the claims. In someinstances, well-known circuits, structures, or techniques have not beenshown or described in detail in order not to obscure the embodiments ofthis disclosure. Like numbers in two or more figures represent the sameor similar elements.

In this description, specific details are set forth describing someembodiments consistent with the present disclosure. Numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments. It will be apparent, however, to one skilled in the artthat some embodiments may be practiced without some or all of thesespecific details. The specific embodiments disclosed herein are meant tobe illustrative but not limiting. One skilled in the art may realizeother elements that, although not specifically described here, arewithin the scope and the spirit of this disclosure. In addition, toavoid unnecessary repetition, one or more features shown and describedin association with one embodiment may be incorporated into otherembodiments unless specifically described otherwise or if the one ormore features would make an embodiment non-functional.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. One of ordinary skill in the art would recognize manyvariations, alternatives, and modifications. Thus, the scope of theinvention should be limited only by the following claims, and it isappropriate that the claims be construed broadly and in a mannerconsistent with the scope of the embodiments disclosed herein.

What is claimed is:
 1. A method comprising: receiving, at a pipelineneural network, an input text of a natural language question and contextthat provides an answer to the natural language question; selecting,using a context selection network in the pipeline neural network, apremium context from the context; determining, using a question typenetwork in the pipeline neural network, a type of the natural languagequestion; and determining the answer to the natural language question asa yes or no answer when the type of the natural language question is ayes or a no question.
 2. The method of claim 1, further comprising:determining, using a span extraction network in the pipeline neuralnetwork, an answer span as the answer to the natural language questionwhen the type of the natural language question is a span question. 3.The method of claim 1, wherein a span extraction network includes a BERTmodel that is trained using a question-paragraph dataset to determine ananswer span for the answer.
 4. The method of claim 1, wherein thecontext selection network includes a classifier neural network forclassifying a first portion of the context as the premium context and asecond portion of the context as a non-premium context.
 5. The method ofclaim 1, wherein the context includes a plurality of paragraphs and thepremium context includes at least two paragraphs.
 6. The method of claim1, wherein the context includes a plurality of paragraphs and thepremium context includes at least one paragraph in the plurality ofparagraphs having a probability above a probability threshold.
 7. Themethod of claim 1, wherein the question type network includes athree-classifier neural network for classifying the type of the naturallanguage question as a yes question, a no question, or a span question.8. The method of claim 1, wherein the context selection network includesa BERT model and a classification layer and further comprising trainingthe classification layer for selecting the premium context from thecontext.
 9. The method of claim 1, wherein the question type networkincludes a BERT model and a classification layer and further comprisingtraining the classification layer for determining the type of thenatural language question.
 10. A system comprising: at least one memoryincluding a pipeline neural network; a processor coupled to the at leastone memory; and the pipeline neural network configured to: receive aninput text of a natural language question and context that provides ananswer to the natural language question; select, using a contextselection network in the pipeline neural network, a premium context fromthe context; determine, using a question type network in the pipelineneural network, that the natural language question is answered by ananswer span; and determine, using a span extraction network in thepipeline neural network and the premium context, the answer span as theanswer to the natural language question.
 11. The system of claim 10,wherein the context selection network includes a classifier neuralnetwork configured to classify a first portion of the context as premiumcontext and a second portion of the context as non-premium context. 12.The system of claim 10, wherein the context includes a plurality ofparagraphs and the premium context includes at least two paragraphs. 13.The system of claim 10, wherein the context includes a plurality ofparagraphs and the premium context includes at least one paragraphhaving a probability above a probability threshold.
 14. The system ofclaim 10, wherein the question type network includes a three-classifierneural network and configured to classify a type of the natural languagequestion as a yes question, a no question, or a span question.
 15. Thesystem of claim 10, wherein the context selection network includes aBERT model and a classification layer and is further configured to trainthe classification layer to select the premium context from the context.16. The system of claim 10, wherein the question type network includes aBERT model and a classification layer and is further configured to trainthe classification layer to determine a type of the answer to thenatural language question.
 17. The system of claim 10, wherein the spanextraction network includes a BERT model that is trained using aquestion-paragraph dataset to determine the answer span for the answer.18. A non-transitory machine-readable medium having stored thereonmachine-readable instructions executable to cause a machine to performoperations that determine an answer to a natural language question, theoperations comprising: receiving, at a pipeline neural network, an inputtext of the natural language question and context that provides theanswer to the natural language question; selecting, using a contextselection network in the pipeline neural network, a premium context fromthe context; determining, using a question type network in the pipelineneural network, a type of the natural language question; and determiningthe answer to the natural language question as a yes or no answer whenthe type of the natural language question is a yes or a no question. 19.The non-transitory machine-readable medium of claim 18, furthercomprising: determining, using a span extraction network in the pipelineneural network, an answer span as the answer to the natural languagequestion when the type of the natural language question is a spanquestion.
 20. The non-transitory machine-readable medium of claim 18,wherein the context selection network includes a classifier neuralnetwork for classifying a first portion of the context as the premiumcontext and a second portion of the context as a non-premium context.